Extract Mark Indicators
Mark Indicators¶
Overview¶
Developer Note: if you may make a PR in the future, be sure to copy this
notebook, and use the gitignore prefix temp to avoid future conflicts.
This is one notebook in a multi-part series on clusterless decoding in Spyglass
- To set up your Spyglass environment and database, see the Setup notebook
- For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook
- Prior to running, please familiarize yourself with the spike sorting pipeline and generate input position data with either the Trodes or DLC notebooks (1, 2, 3).
The goal of this notebook is to populate the UnitMarksIndicator table, which depends on a series of tables in the spike sorting pipeline:
SpikeSorting->CuratedSpikeSorting->UnitMarks->UnitMarkIndicators
While clusterless decoding avoids actual spike sorting and curation, we need to pass through these tables to maintain (relative) pipeline simplicity. Pass-through tables keep spike sorting and clusterless mark extraction as similar as possible, by using shared steps. Here, "spike sorting" involves simple thresholding (sorter: clusterless_thresholder). The populate_mark_indicators will run each of these steps provided we have data in SpikeSortingSelection and IntervalPositionInfo.
SpikeSortingSelection depends on:
SpikeSortingRecordingSpikeSorterParametersArtifactRemovedIntervalList.
SpikeSortingRecording depends on:
SortGroupSortIntervalSpikeSortingPreprocessingParametersLabTeam
Imports¶
import os
import datajoint as dj
from pprint import pprint
# change to the upper level folder to detect dj_local_conf.json
if os.path.basename(os.getcwd()) == "notebooks":
os.chdir("..")
dj.config.load("dj_local_conf.json") # load config for database connection info
import spyglass.common as sgc
import spyglass.spikesorting as sgs
import spyglass.decoding as sgd
import spyglass.utils as sgu
# ignore datajoint+jupyter async warnings
import warnings
warnings.simplefilter("ignore", category=DeprecationWarning)
warnings.simplefilter("ignore", category=ResourceWarning)
warnings.simplefilter("ignore", category=UserWarning)
[2023-08-02 18:58:19,324][INFO]: Connecting root@localhost:3306 [2023-08-02 18:58:19,360][INFO]: Connected root@localhost:3306 /home/cb/miniconda3/envs/spy/lib/python3.9/site-packages/spikeinterface/sortingcomponents/peak_detection.py:643: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details. @numba.jit(parallel=False) /home/cb/miniconda3/envs/spy/lib/python3.9/site-packages/spikeinterface/sortingcomponents/peak_detection.py:668: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details. @numba.jit(parallel=False) Cupy is not installed or GPU is not detected. Ignore this message if not using GPU
Select Data¶
nwb_file_name = "J1620210531.nwb"
nwb_copy_file_name = sgu.nwb_helper_fn.get_nwb_copy_filename(nwb_file_name)
Spike Sorting Selection¶
We can investigate the populate_mark_indicators function we want to use with
?:
?sgd.clusterless.populate_mark_indicators
/home/edeno/miniconda3/envs/spyglass/lib/python3.8/site-packages/seaborn/rcmod.py:82: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(mpl.__version__) >= "3.0": /home/edeno/miniconda3/envs/spyglass/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. other = LooseVersion(other)
Connecting edeno@lmf-db.cin.ucsf.edu:3306
Signature: populate_mark_indicators( spikesorting_selection_keys: list, mark_param_name='default', position_info_param_name='default_decoding', ) Docstring: <no docstring> File: /stelmo/edeno/nwb_datajoint/src/spyglass/decoding/clusterless.py Type: function
From the docstring, we see that we need to supply spikesorting_selection_keys,
which is a list of SpikeSorting keys as dictionaries. We provide a list to
extract marks for many electrode at once.
Here are the primary keys required by SpikeSorting:
sgs.SpikeSorting.primary_key
['nwb_file_name', 'sort_group_id', 'sort_interval_name', 'preproc_params_name', 'team_name', 'sorter', 'sorter_params_name', 'artifact_removed_interval_list_name']
Here is an example of what spikesorting_selection_keys should look like:
spikesorting_selections = [
{
"nwb_file_name": "J1620210531_.nwb",
"sort_group_id": 0,
"sort_interval_name": "raw data valid times no premaze no home",
"preproc_params_name": "franklab_tetrode_hippocampus",
"team_name": "JG_DG",
"sorter": "clusterless_thresholder",
"sorter_params_name": "clusterless_fixed",
"artifact_removed_interval_list_name": "J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times",
},
{
"nwb_file_name": "J1620210531_.nwb",
"sort_group_id": 1,
"sort_interval_name": "raw data valid times no premaze no home",
"preproc_params_name": "franklab_tetrode_hippocampus",
"team_name": "JG_DG",
"sorter": "clusterless_thresholder",
"sorter_params_name": "clusterless_fixed",
"artifact_removed_interval_list_name": "J1620210531_.nwb_1_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times",
},
]
WARNING: This process relies on both SpikeSortingSelection and
IntervalPositionInfo. You can check your database with the following:
sgs.SpikeSortingSelection & {
"nwb_file_name": nwb_copy_file_name,
"sorter": "clusterless_thresholder",
}
| nwb_file_name name of the NWB file | sort_group_id identifier for a group of electrodes | sort_interval_name name for this interval | preproc_params_name | team_name | sorter | sorter_params_name | artifact_removed_interval_list_name | import_path optional path to previous curated sorting output |
|---|---|---|---|---|---|---|---|---|
| J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | |
| J1620210531_.nwb | 1 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_1_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times |
Total: 2
Remember to replace the nwb_file_name with your own nwb file name
sgc.IntervalPositionInfo & {
"nwb_file_name": nwb_copy_file_name,
"position_info_param_name": "default_decoding",
}
| position_info_param_name name for this set of parameters | nwb_file_name name of the NWB file | interval_list_name descriptive name of this interval list | analysis_file_name name of the file | head_position_object_id | head_orientation_object_id | head_velocity_object_id |
|---|---|---|---|---|---|---|
| default_decoding | J1620210531_.nwb | pos 0 valid times | J1620210531_XOKZ4G53LE.nwb | 200c54a9-57dc-4ed4-8e48-85cc9387b922 | 26f4ec71-2f5f-4ef2-82f1-ee3a89048168 | 7f9ffe2f-31e5-47ac-bad9-558440cd5f1d |
| default_decoding | J1620210531_.nwb | pos 1 valid times | J1620210531_B257TS35XJ.nwb | c164cdfd-c328-49b2-b6b4-f8663a17f5d6 | e1f26617-c44b-4464-942d-5e6755606789 | f3c69ac6-5f0a-4622-827b-44a769ce4f77 |
| default_decoding | J1620210531_.nwb | pos 10 valid times | J1620210531_1RICS57YYG.nwb | 2394ec2d-c076-4b11-a40b-91e4b51b81fa | 005080ee-b93f-4b3c-bcd0-595b46fedab2 | 47158c4f-3632-4ce1-9f1e-043f1ea5a5ec |
| default_decoding | J1620210531_.nwb | pos 11 valid times | J1620210531_HZSI1BGTE7.nwb | 295aa187-81cd-4209-b96d-163f12a716ed | 19a95d85-8cb9-4e06-82ee-26bc6df7c8c8 | 9abb39ce-f0e2-4637-b9e9-5f7b6891201d |
| default_decoding | J1620210531_.nwb | pos 12 valid times | J1620210531_M0UMC7TZ49.nwb | ae4c015c-9d5f-4ca3-bbcd-049c1e1ae50b | 1bbc244c-4d85-4e63-9d69-10815e37276c | be8f6ba5-3cc9-4ac3-96af-5c3e84b9851c |
| default_decoding | J1620210531_.nwb | pos 13 valid times | J1620210531_TREA1SOT03.nwb | c4bfe562-f523-495c-99e5-681524661264 | 65023b5e-321b-4d6e-b975-ec03cbd46247 | ce391291-6107-420a-a73d-9a18d343a47a |
| default_decoding | J1620210531_.nwb | pos 14 valid times | J1620210531_93PXV7F4CX.nwb | 8521f813-635d-41cd-8f10-b00f26a7f957 | 75d08124-57f7-481b-ae44-0752a5912c18 | 84ce4054-8c11-49f2-8295-7d8173cb7eb7 |
| default_decoding | J1620210531_.nwb | pos 15 valid times | J1620210531_FUJ1B1OKOA.nwb | e72e5fcd-3d80-463a-a9d1-305e6e99f8db | 6e8dc0e2-ab02-4305-bf2e-e2d208bae77a | b7589719-49ba-48e7-a20b-88a911c3f8a5 |
| default_decoding | J1620210531_.nwb | pos 16 valid times | J1620210531_76HL0SNP59.nwb | b3124712-55db-4b0b-9bdc-77f03a0b8c98 | 08bfd82c-1e7c-4aa7-bd9e-256577b26e9c | 85ffd435-3dc4-49ab-adea-a3327817a51d |
| default_decoding | J1620210531_.nwb | pos 17 valid times | J1620210531_5F8VRT0OC4.nwb | 78b8f7e4-fb90-438d-8551-d15a4d97484a | 7b96d15c-9721-44ea-9ecd-d8a2b60711d6 | 12336768-227e-4c0c-9ade-b3c7ac6c2de3 |
| default_decoding | J1620210531_.nwb | pos 18 valid times | J1620210531_PP7XK1G26S.nwb | 39ab3238-a0dd-4906-95c5-8e534458e371 | ca0f939d-f0a7-4df4-9946-2f605ef5ab76 | 571277d0-885e-4d0e-b802-6ad8047ee3f1 |
| default_decoding | J1620210531_.nwb | pos 19 valid times | J1620210531_AQZHSP2IDM.nwb | b5a21053-3239-4943-aee1-3227b4915d5f | 9f0c61c8-90ed-4a74-873c-e250aa872a2a | 714f6ceb-6d70-474c-8c42-6f9fa3ff77e1 |
...
Total: 21
Populate Mark Indicators¶
Now that we've checked those, we can run the function:
sgd.clusterless.populate_mark_indicators(spikesorting_selections)
We can verify that this worked:
plt.plot(position_info.head_position_x, position_info.head_position_y)
| curation_id a number correponding to the index of this curation | nwb_file_name name of the NWB file | sort_group_id identifier for a group of electrodes | sort_interval_name name for this interval | preproc_params_name | team_name | sorter | sorter_params_name | artifact_removed_interval_list_name | mark_param_name a name for this set of parameters | interval_list_name descriptive name of this interval list | sampling_rate | analysis_file_name name of the file | marks_indicator_object_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 0 valid times | 500.0 | J1620210531_NS19JB65RU.nwb | d0de79cc-f2be-4e86-b4e8-e58a8aef1c02 |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 1 valid times | 500.0 | J1620210531_YP9MXAPE2G.nwb | 7598dc69-a024-4140-8b37-d1342a0e7829 |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 10 valid times | 500.0 | J1620210531_AGOI7DXRQX.nwb | 12a60beb-3ac2-41ae-a091-eebd0b14672c |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 11 valid times | 500.0 | J1620210531_VXGQCG8UYO.nwb | d0d40c96-e5a1-494f-b8a7-bf475304d640 |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 12 valid times | 500.0 | J1620210531_NF8RWZGCSG.nwb | 2b6476a7-143f-4760-bafa-e4959f0e735a |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 13 valid times | 500.0 | J1620210531_F7OHK7UMO3.nwb | 84aea599-dbf1-4d11-bca2-00aa6ae9967a |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 14 valid times | 500.0 | J1620210531_MPXHIH43Y1.nwb | 78224e49-0118-458d-9aee-727db98fc306 |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 15 valid times | 500.0 | J1620210531_YFG1894358.nwb | 721526f8-df6d-442e-84ee-bf7dd6ff8c19 |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 16 valid times | 500.0 | J1620210531_VJ5V0LRJKU.nwb | 39258f97-4508-4068-9b2b-ce419a5d718d |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 17 valid times | 500.0 | J1620210531_NACXLH30KV.nwb | bf312f88-af31-4247-83dc-9633d191401a |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 18 valid times | 500.0 | J1620210531_LFKSWYVVRS.nwb | af89e3e1-71f7-4c7c-a5ef-94d3cc7a1203 |
| 0 | J1620210531_.nwb | 0 | raw data valid times no premaze no home | franklab_tetrode_hippocampus | JG_DG | clusterless_thresholder | clusterless_fixed | J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times | default | pos 19 valid times | 500.0 | J1620210531_R9V8JD4IQA.nwb | 3616a7b7-fca2-43bb-9c72-44eb3c22af80 |
...
Total: 42
position_info.shape, marks.shape
((655645, 6), (655645, 4, 22))
from spyglass.common.common_interval import interval_list_intersect
from spyglass.common import IntervalList
key = {}
key["interval_list_name"] = "02_r1"
key["nwb_file_name"] = nwb_copy_file_name
interval = (
IntervalList
& {
"nwb_file_name": key["nwb_file_name"],
"interval_list_name": key["interval_list_name"],
}
).fetch1("valid_times")
valid_ephys_times = (
IntervalList
& {
"nwb_file_name": key["nwb_file_name"],
"interval_list_name": "raw data valid times",
}
).fetch1("valid_times")
position_interval_names = (
IntervalPositionInfo
& {
"nwb_file_name": key["nwb_file_name"],
"position_info_param_name": "default_decoding",
}
).fetch("interval_list_name")
valid_pos_times = [
(
IntervalList
& {
"nwb_file_name": key["nwb_file_name"],
"interval_list_name": pos_interval_name,
}
).fetch1("valid_times")
for pos_interval_name in position_interval_names
]
intersect_interval = interval_list_intersect(
interval_list_intersect(interval, valid_ephys_times), valid_pos_times[0]
)
valid_time_slice = slice(intersect_interval[0][0], intersect_interval[0][1])
valid_time_slice
slice(1581886916.3153033, 1581888227.5987928, None)
from replay_trajectory_classification import ClusterlessClassifier
from replay_trajectory_classification.environments import Environment
from replay_trajectory_classification.continuous_state_transitions import (
RandomWalk,
Uniform,
)
from spyglass.decoding.clusterless import ClusterlessClassifierParameters
marks = marks.sel(time=valid_time_slice)
position_info = position_info.loc[valid_time_slice]
parameters = (
ClusterlessClassifierParameters()
& {"classifier_param_name": "default_decoding_gpu"}
).fetch1()
parameters["classifier_params"]["clusterless_algorithm_params"] = {
"mark_std": 24.0,
"position_std": 3.0,
"block_size": int(2**13),
"disable_progress_bar": False,
"use_diffusion": False,
}
parameters["classifier_params"]["environments"][0] = Environment(
place_bin_size=3.0
)
import cupy as cp
with cp.cuda.Device(0):
classifier = ClusterlessClassifier(**parameters["classifier_params"])
classifier.fit(
position=position_info[["head_position_x", "head_position_y"]].values,
multiunits=marks.values,
**parameters["fit_params"],
)
results = classifier.predict(
multiunits=marks.values,
time=position_info.index,
**parameters["predict_params"],
)
logging.info("Done!")
from spyglass.decoding.visualization import (
create_interactive_2D_decoding_figurl,
)
view = create_interactive_2D_decoding_figurl(
position_info,
marks,
results,
classifier.environments[0].place_bin_size,
position_name=["head_position_x", "head_position_y"],
head_direction_name="head_orientation",
speed_name="head_speed",
posterior_type="acausal_posterior",
sampling_frequency=500,
view_height=800,
)
Up Next¶
Next, we'll start the process of decoding representations of position with ephys data. This can be done either with GPUs or clusterless.